Kullback–Leibler divergence

$$D_\text{KL}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right).$$

This captures the difference between the entropy of $P(x)$ and the “wrong” entropy of $P(x)$ measured with $Q(x)$ ($P(x) \log P(x) - P(x) \log Q(x)$).

A common way to measure “distance” between two distributions. It’s not a real distance because it’s not symmetric. The symmetrized version is called Jensen–Shannon divergence.

See also Gibbs’ inequality